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Spurred  by  the  piecemeal  approach  to  school  reform 
that  had  produced  little  change  in  student  performance. 
New  American  Schools  (NAS),  a  private  nonprofit  organi¬ 
zation,  launched  its  efforts  for  whole-school  reform  in 
1991.  NAS's  mission  was  to  help  schools  and  districts 
significantly  raise  the  achievement  of  large  numbers  of 
students  through  the  use  of  whole-school  designs. 

NAS's  core  premise  was  that  all  high-quality  schools 
possess,  de  facto,  a  unifying  design  that  enables  all  staff 
members  to  function  to  the  best  of  their  abilities  and  that 
integrates  research-based  practices  into  a  coherent  and 
mutually  reinforcing  set  of  effective  approaches  to  teaching 
and  learning.  The  best  way  to  ensure  that  lower-perform¬ 
ing  schools  adopted  successful  designs  was  to  fund  teams 
to  develop  "break  the  mold"  school  designs  that  could 
readily  be  adopted  by  communities  nationwide.  After 
developing  the  designs,  the  teams  would  then  implement 
them  in  schools  throughout  the  country.  This  adoption 
process  would  fulfill  NAS's  primary  goal  of  improving  the 
performance  of  large  numbers  of  students. 

This  whole-school  design  approach  to  better  educa¬ 
tional  outcomes  was  a  dramatically  different  way  of 
initiating  and  disseminating  large-scale  educational 
improvements  to  many  schools  across  the  country.  It  was 
a  unique  combination  of  (1)  private  sector  involvement 
using  a  venture  capitalist  approach;  (2)  the  choice  of 
whole-school  design  as  a  vehicle  for  reform;  and  (3)  the 
ambitious  goal  of  achieving  scale-up  (disseminating 
designs  to  a  large  number  of  schools  in  a  jurisdiction  that 
agreed  to  partner  with  NAS). 

The  NAS  experimental  approach  required  careful 
development  and  demonstration  of  designs  before  moving 
to  the  scale-up  stage;  therefore,  NAS  proceeded  in  phases: 

•  Competition  and  selection  phases  (1992) 

•  One-year  development  phase  (1992-1993) 


•  Two-year  demonstration  phase  (1993-1995,  including 
the  1993-1994  and  1994-1995  school  years) 

•  Three-year  scale-up  phase  (1995-1998,  from  the  school 
year  starting  in  1995  to  the  school  year  ending  in  1998). 

RAND’S  ROLE  IN  THE  NAS  INITIATIVE 

NAS  asked  RAND  to  provide  analytic  support  to  its 
school  reform  efforts.  This  support  took  many  forms  but 
primarily  was  intended  to  document  and  analyze  the  con¬ 
ditions  under  which  NAS  progressed  toward  its  goals  of 
widespread  implementation  of  its  designs  and  improved 
student  performance  associated  with  the  use  of  those 
designs.  The  support  from  RAND  included  the  following 
tasks: 

•  Documenting  NAS's  efforts  to  assess  its  contributions  to 
education  reform 

•  Describing  the  designs  and  analyzing  changes  in  them 
over  time 

•  Assessing  the  level  of  implementation  in  design-based 
schools  during  the  demonstration  and  scale-up  phases 

•  Identifying  factors  that  impede  or  encourage  implemen¬ 
tation  in  the  demonstration  and  scale-up  phases 

•  Measuring  whether  the  adoption  of  school  designs 
resulted  in  the  outcomes  desired  by  NAS  and  its  partner¬ 
ing  districts  in  the  scale-up  phase. 

These  tasks  were  conducted  over  the  first  seven  years 
of  the  NAS  initiative,  from  the  demonstration  phase 
through  the  scale-up  phase,  and  the  results  are  document¬ 
ed  in  a  series  of  RAND  reports.  Over  this  period,  RAND's 
program  of  studies  has  included  the  following: 

•  A  longitudinal  sample  of  more  than  100  NAS  schools 
that  began  implementation  early  on  in  the  scale-up 
phase,  including  data  on  implementation  and  perfor¬ 
mance  from  principals,  teachers,  and  districts 


•  Case  studies  in  40  schools  to  analyze  implementation 
and  the  role  that  districts  play  in  impeding  or  enabling 
comprehensive  school  reform 

•  A  description  of  how  designs  have  evolved  from  the 
initial  proposal  stage  to  implementation  at  scale-up  in 
schools  across  the  nation 

•  Analyses  in  one  urban  school  district  of  how  whole- 
school  designs  promote  changes  in  classroom  instruc¬ 
tion,  teaching  and  learning,  and  individual-level  student 
achievement  scores 

•  An  analysis  of  performance  differences  among  high- 
implementing  NAS  sites  and  ongoing  discussions  with 
NAS  staff  and  design  team  leaders. 

Each  piece  of  the  RAND  program  of  studies  supplied 
an  important  building  block  toward  a  full  understanding 
of  the  NAS  initiative.  Together,  these  studies  provide  a 
cogent,  consistent,  and  comprehensive  examination  of  the 
NAS  initiative  and  offer  useful  and  timely  information  to 
decisionmakers  considering  or  already  engaging  in  whole- 
school  reform.  The  final  report  in  the  series  (Berends, 
Bodilly,  and  Kirby,  2002)  takes  a  retrospective  look  at  the 
NAS  initiative,  drawing  together  the  findings  from 
RAND's  many  studies  and  reflecting  on  the  lessons 
learned  from  this  initiative  and  the  implications  for  future 
school  reform  efforts.  This  latest  report  should  interest 
policymakers  and  practitioners  in  K~12  education  because 
it  offers  some  important  lessons  on  attempts  to  improve 
student  performance  through  external  interventions — the 
focus  of  the  current  federally  funded  Comprehensive 
School  Reform  Demonstration  program. 

A  BRIEF  HISTORY  OF  NAS  FOCUSING  ON  ITS 
THEORY  OF  ACTION 

Any  educational  reform  must  have  two  components:  a 
theory  oflearnwg  (that  brings  together  assumptions  about 
how  students  learn,  instructional  strategies,  and  perfor¬ 
mance)  and  a  theory  of  action  (that  focuses  on  the  conditions 
under  which  the  reform  will  work).  The  theory  of  learning 
in  the  NAS  initiative  was  embodied  in  the  individual 
designs. 

Initially,  NAS's  theory  of  action  was  quite  simple  and 
undeveloped:  Designs  would  be  developed,  schools  would 
adopt  them  in  some  unspecified  manner,  and  this  adoption 
would  improve  student  outcomes.  NAS  helped  to  create 
the  designs  through  a  request  for  proposals  process  from 
October  1991  to  July  1992  during  which  more  than  600 
teams  applied.  NAS  chose  11  design  teams  for  initial 
development.  From  July  1992  to  July  1993,  those  teams 
worked  to  further  develop  their  theories  of  learning.  A 
year  later,  NAS  reduced  the  number  of  teams  from  11  to  9 
in  part  due  to  funding  difficulties.  The  two  teams  that  NAS 


removed  had  a  district  as  their  team  leader.  This  decision 
indicated  that  NAS's  theory  of  action  did  not  include 
locally  based  and  led  teams,  in  part  because  of  its  ambition 
for  national  scale-up. 

The  demonstration  phase,  from  1993  to  1995,  took 
place  in  147  schools  that  the  design  teams  selected  to  part¬ 
ner  with  them.  RAND's  analysis  of  the  progress  at  that 
point  indicated  that  an  important  component  of  the  design 
team  intervention  was  the  assistance  that  teams  provided 
to  schools,  which  enabled  those  schools  to  adopt  the 
designs.  NAS  calls  this  component  of  the  initiative 
"design-based  assistance."  NAS's  theory  of  change 
evolved  to  include  the  notion  that  design  teams  would 
provide  not  just  designs  but  also  design-based  assistance 
to  implement  those  designs  successfully. 

RAND's  analysis  of  the  progress  in  the  demonstration 
schools  also  showed  that  school  and  district  factors  had  a 
strong  relationship  to  the  success  of  an  implementation. 
NAS  by  then  understood  that  to  succeed  it  would  have  to 
ensure  a  supportive  environment  for  its  designs.  During 
the  demonstration  phase,  NAS  removed  two  more  teams, 
both  of  which  did  not  have  national  scale-up  intentions 
and  which  appeared  at  the  time  to  have  characteristics 
that  were  specific  to  districts  or  states. 

NAS  outlined  its  scale-up  strategy — a  third  iteration 
of  its  theory  of  action.  Because  of  its  experiences  during 
the  demonstration  phase,  NAS  became  more  aware  of  the 
importance  of  gaining  district-level  support  for  teams  and 
providing  a  supportive  district  structure,  including 
resources,  school  autonomy,  and  professional  develop¬ 
ment  for  the  schools.  Thus,  it  chose  a  "concentration" 
strategy — transforming  a  few  districts  by  having  a  signifi¬ 
cant  number  of  schools  within  those  districts  adopt 
design-based  assistance. 

Ten  jurisdictions  agreed  to  partner  with  NAS.  Those 
jurisdictions  promised  to  get  approximately  30  percent  of 
their  schools  to  use  the  designs  within  a  three-year  period 
and  to  provide  a  supportive  environment  for  the  schools' 
efforts.  The  30-percent  figure  was  somewhat  arbitrary. 

The  theory  of  action  was  that  if  NAS  could  implement  its 
designs  in  about  a  third  of  a  district's  schools,  the  district 
itself  would  have  to  change  its  policies  to  become  perma¬ 
nently  more  supportive  of  its  schools.  In  addition,  NAS 
insisted  that  the  design  teams  should  become  more  self- 
sufficient  at  this  point.  Design  teams  would  charge  the  dis¬ 
tricts  fees  for  the  design-based  assistance,  and  NAS  would 
work  with  districts  to  help  make  a  supportive  environ¬ 
ment  evolve.  NAS  promised  districts  that  that  by  using  its 
designs,  district  schools  would  be  able  to  show  dramatic 
improvements  on  standardized  district-mandated  tests 
within  the  three-year  period. 
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The  seven  teams  entering  the  scale-up  phase  included 
the  following: 

•  Authentic  Teaching,  Learning,  and  Assessment  for  All 
Students  (ATLAS) 

•  Audrey  Cohen  College  (AC)  (currently  renamed 
Purpose-Centered  Education) 

•  Co-NECT  Schools  (CON) 

•  Expeditionary  Learning  Outward  Bound  (EL) 

•  Modern  Red  Schoolhouse  (MRSH) 

•  National  Alliance  for  Restructuring  Education 
(NARE)  (currently  renamed  America's  Choice  Design 
Network) 

•  Roots  &  Wings  (RW). 

NAS  partnered  with  the  following  jurisdictions: 
Cincinnati;  Dade  County,  Florida;  Kentucky;  Maryland; 
Memphis;  Philadelphia;  Pittsburgh;  San  Antonio;  San 
Diego;  and  five  districts  in  the  state  of  Washington.  Close 
to  185  schools  partnered  with  design  teams  in  these  juris¬ 
dictions.  As  a  whole,  NAS  design  teams  spread  to  more 
than  550  schools  by  1995. 

CONCEPTUAL  FRAMEWORK  OF  THE  STUDIES 

The  conceptual  framework  underpinning  the  RAND 
studies  of  NAS  draws  from  previous  research  on  imple¬ 
mentation  of  school  reforms  and  educational  change.  A 
critical  assumption  underlying  the  designs  is  that  coher¬ 
ent,  focused,  and  sustained  implementation  of  key  design 
components  (including  professional  development, 
curriculum  and  instructional  materials,  content  and 
performance  standards,  assessments,  organization  and 
governance,  and  parent  and  community  involvement) 
will  eventually  change  school  and  classroom  learning 
environments  and  thereby  students'  academic  outcomes. 
Implementation  consists  of  the  process  of  putting  into 
practice  the  elements  or  sets  of  activities  defined  by  design 
teams  as  core  components  of  their  designs. 

However,  throughout  the  history  of  educational 
reform  efforts,  a  consistent  theme  has  been  that  the  pro¬ 
cess  of  planned  educational  change  is  much  more  complex 
than  what  was  initially  anticipated.  Past  research  on 
external  change  agents  as  reform  mechanisms  in  K-12 
education  has  shown  that  as  these  externally  developed 
interventions  are  implemented,  they  tend  to  change  sig¬ 
nificantly  as  they  adapt  to  local  conditions  and  contexts  or 
progress  through  scaling  up.  Often,  those  conducting  the 
intervention  develop  strategies  to  assist  schools  in  under¬ 
standing  and  implementing  the  intervention.  Despite 
these  measures,  implementation  tends  to  vary  across  sites, 
and  the  outcomes — in  terms  of  the  desired  change — also 
vary  considerably.  This  variation  in  outcomes  occurs 


largely  because  of  the  number  of  players  involved  and  the 
number  of  factors  that  need  to  be  in  alignment  to  support 
such  fundamental  change. 

The  factors  that  could  be  expected  to  affect  both  imple¬ 
mentation  and  outcomes  include  the  following,  some  of 
which  were  not  readily  controlled  by  NAS  or  its  design 
teams: 

•  The  design  itself  and  its  ability  to  offer  coherent,  compre¬ 
hensive,  and  consistent  education  programs,  and  the 
assistance  offered  to  schools  by  the  design  teams  to 
ensure  implementation 

•  The  efficacy  of  the  selection  and  matching  of  designs 
and  schools  to  ensure  that  teachers  "buy  in"  to  the 
design 

•  The  capacity  of  the  specific  schools  for  undertaking  the 
reform,  including  the  schools'  other  efforts  at  reform, 
educational  leadership,  and  teaching  capability 

•  School-specific  demographics,  enrollment  and  grade 
level  of  the  schools,  and  climate 

•  The  context  set  by  the  district,  including  the  existing 
infrastructure  support  and  incentives  for  design  imple¬ 
mentation  and  improved  student  performance 

•  Other  factors,  such  as  the  context  set  by  state  policies  for 
testing  and  accountability,  community  influences,  and 
NAS  funding  policies. 

FINDINGS 

Here  we  summarize  the  specific  findings  of  the  various 
analyses.  The  findings  are  grouped  into  three  broad  areas: 
(1)  the  evolution  of  designs,  (2)  the  level  of  implementation 
observed  in  the  NAS  sites  and  the  factors  related  to  imple¬ 
mentation,  and  (3)  student  outcomes. 

Overview  of  Changes  to  Designs  over  Time 

These  findings  are  based  on  a  continuous  tracking  of 
the  designs  and  design  teams  throughout  the  analytic  effort 
(Bodilly,  2001).  Consistent  with  the  literature  on  external 
change  agents,  the  designs  adapted  to  local  conditions  over 
time.  Some  of  the  design  developments  were  beneficial  for 
enabling  schools  to  improve.  For  example,  the  growth  in 
assistance  packages,  the  further  development  of  curricular 
units,  and  the  development  of  protocols  for  a  school's 
choice  of  design  all  appeared  to  be  positive  adaptations  of 
the  designs.  Development  of  basic  skills  curriculum  could 
also  be  considered  a  positive  adaptation  when  well  inte¬ 
grated  with  the  principles  of  the  design  and  not  simply  a 
quick  add-on  to  meet  district  demands. 

Other  changes  to  the  designs,  while  understandable, 
became  problematic.  Adaptation  to  district  and  school  poli¬ 
cies  led  to  some  designs  accepting  an  unaligned  and  inco- 
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herent  mix  of  standards,  assessments,  curriculum,  instruc¬ 
tion,  and  professional  development.  This  situation  also 
allowed  for  considerable  local  adaptation  to  the  point  that 
one  might  question  the  unifying  or  coherent  nature  of  a 
design. 

Because  changes  were  occurring  to  some  designs  over 
the  demonstration  and  scale-up  phases,  a  major  compo¬ 
nent  of  NAS's  theory  of  action — a  coherent  and  unifying 
design — was  often  missing  or  was  constantly  undergoing 
revision.  It  cannot  be  emphasized  enough  that  during  the 
entire  period  of  the  RAND  studies,  the  designs  were  still 
in  a  state  of  development  and  evolution.  Consistent  with 
the  literature,  implementation  assistance  (that  is,  design- 
based  assistance)  became  an  important  part  of  the  inter¬ 
vention  as  time  went  on. 

Overview  of  the  Implementation  Findings 

RAND  studied  the  implementation  of  designs  in  both 
the  demonstration  and  scale-up  phases.  The  purpose  of 
the  studies  was  to  measure  the  level  of  implementation  in 
schools  and  to  determine  the  conditions  under  which 
implementation  prospered  (Bodilly,  1998;  Keltner,  1998; 
Bodilly  and  Berends,  1999;  Berends,  2000;  Berends  et  al, 
2001;  Kirby  et  al,  2001;  and  Berends,  Bodilly,  and  Kirby, 
2002). 

The  demonstration  schools  showed  some  promise, 
but  RAND  identified  many  barriers  to  implementation 
(Bodilly  et  al.,  1995;  Mitchell,  1996).  During  scale-up, 

NAS  and  the  design  teams  partnered  with  schools  and 
districts  that  were  characterized  by  a  host  of  problems 
related  to  poverty,  achievement,  and  school  and  district 
characteristics  (Berends,  1999),  Achieving  high  levels  of 
implementation  within  those  schools  across  all  teachers 
proved  to  be  challenging.  The  case  study  analyses  found 
that  by  two  years  into  implementation  approximately  half 
of  the  sample  sites  were  implementing  designs  at  a  level 
consistent  with  the  expectations  of  NAS  and  the  design 
teams.  The  other  half  were  implementing  at  a  rate  below 
that  level.  All  sites  reported  many  barriers  to  further 
implementation. 

The  longitudinal  sample  of  teachers  supported  the 
findings  of  the  case  studies  that  showed  weak  implemen¬ 
tation  and  stagnation: 

•  For  the  entire  sample  of  schools  RAND  surveyed, 
implementation  in  the  scale-up  schools  increased  mod¬ 
estly  from  1997  to  1999.  The  between-school  variance  in 
the  level  of  implementation  decreased  somewhat  over 
time,  but  the  within-school  variance  increased.  There 
was  much  greater  variance  in  implementation  within 
schools  than  between  schools,  suggesting  that  designs 
had  not  become  "schoolwide." 


•  There  were  large  differences  in  implementation  by  juris¬ 
diction,  by  design,  and  across  schools. 

•  For  schools  newly  adopting  designs,  implementation 
increased  and  expanded  over  the  first  four  years  after 
the  schools  adopted  designs,  although  at  decreasing 
rates  each  year. 

As  expected,  many  factors  influenced  the  level  of 
implementation,  and  the  various  RAND  analyses  identified 
similar  conditions  that  fostered  higher  levels  of  implemen¬ 
tation. 

Designs  and  Design-Based  Assistance.  Implementa¬ 
tion  varied  by  design.  In  the  longitudinal  survey  sample, 
schools  with  CON,  NARE,  and  RW  designs  clearly  report¬ 
ed  higher  implementation  levels.  The  high  levels  of  imple¬ 
mentation  reported  in  all  the  studies  were  related  to  clear 
communication  with  and  strong  assistance  from  the  design 
teams.  In  the  last  of  the  case  study  analyses,  which  used  a 
small  sample  of  schools,  we  found  that  several  design 
teams  did  not  have  a  large  amount  of  information  about 
their  schools  nor  had  they  adequately  measured  the  level 
of  implementation  at  the  schools.  Taken  together,  the  evi¬ 
dence  shows  that  several  design  teams  need  to  make 
significant  improvements  to  ensure  the  implementation 
of  their  designs. 

Selection  Process.  A  case  study  in  Bodilly  (1998) 
revealed  that  those  school  administrators  and  teachers  who 
felt  that  their  choices  were  well  informed  reported  higher 
levels  of  implementation  at  their  schools  than  those  who 
reported  either  being  forced  to  accept  a  design  or  not 
understanding  the  nature  of  the  design.  In  the  longitudinal 
survey  sample,  teacher  support  and  buy-in  were  related  to 
higher  levels  of  implementation  and  vice  versa.  The  case 
study  work  in  Bodilly  (1998)  also  revealed  that  school  prin¬ 
cipals  played  an  important  role  in  ensuring  both  a  sound 
selection  process  and  teacher  buy-in. 

School  Capacity.  School  capacity  was  important  in 
supporting  implementation.  The  leadership  of  a  principal 
was  a  key  contributor  to  implementation  across  all  the 
studies.  The  survey  analyses  indicated  that  teachers 
viewed  a  principal's  leadership  as  the  most  important  pre¬ 
dictor  of  the  implementation  level  achieved.  The  teacher 
survey  also  indicated  that  teachers'  perceptions  of  students 
and  their  readiness  to  learn  were  both  significantly  related 
to  the  level  of  implementation.  The  various  implementa¬ 
tion  case  studies  indicated  that  many  other  reforms  were 
taking  place  in  the  schools  while  the  NAS  reform  was 
under  way.  In  many  instances,  the  numerous  reforms  over¬ 
loaded  teachers  and  reduced  their  capacity  to  implement 
the  designs. 

School  Context.  The  teacher  sur\^eys  revealed  that 
implementation  was  higher  in  high-poverty  schools  and 
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schools  serving  large  numbers  of  minority  students. 
However,  in  schools  that  served  significant  numbers  of 
both  poor  and  minority  students,  implementation  levels 
were  significantly  lower.  Across  our  implementation  stud¬ 
ies,  we  found  that  implementation  was  higher  in  elemen¬ 
tary  schools  than  in  secondary  schools  and  was  higher  in 
smaller  schools  than  in  larger  ones. 

District  Context.  There  were  large  differences  in 
implementation  among  jurisdictions  across  all  of  our  stud¬ 
ies.  In  general,  implementation  was  higher  in  those  dis¬ 
tricts  that  (1)  were  more  supportive  of  the  NAS  designs 
and  were  characterized  as  having  a  stable  district  leader¬ 
ship  that  backed  the  reform  and  made  it  central  to  its 
improvement  efforts;  (2)  were  not  burdened  with  crises 
such  as  budget  shortfalls  or  redistricting;  (3)  had  a  coher¬ 
ent  program  of  reform;  (4)  had  resources  dedicated  to  the 
implementation  effort;  (5)  had  significant  school-level 
autonomy;  and  (6)  had  trusting  relationships  among 
school,  district,  and  union  staff. 

Several  case  study  analyses  highlighted  significant 
barriers  to  implementation  that  were  inherent  in  district 
and  union  policies.  The  San  Antonio  classroom  study  gave 
particular  insight  into  how  district-level  behaviors  affected 
schools  and  their  willingness  to  undertake  and  implement 
designs  (Berends  et  al.,  2002),  Teachers'  views  of  the  con¬ 
sistency  and  effectiveness  of  district  leadership  proved  to 
be  positively  associated  with  the  teachers'  efforts  toward 
implementation.  However,  multiple  reforms  easily  over¬ 
whelmed  teachers  in  their  efforts  at  adopting  a  new  school 
design.  Most  disruptive  was  the  high-stakes  state  testing 
regime,  which  inadvertently  encouraged  a  specific  focus 
on  basic  skills  and  which  resulted  in  the  district  adopting 
specific  curricular  programs  in  addition  to  implementing 
the  designs.  Unfortunately,  these  curricular  programs  con¬ 
flicted  with  the  designs  and  resulted  in  lower  levels  of 
implementation. 

Related  RAND  research  and  exit  interviews  with  the 
principals  of  schools  that  had  dropped  the  NAS  designs 
highlighted  the  importance  of  adequate  funding.  Lack  of 
funding  was  the  single  most  important  reason  cited  by 
most  schools  in  their  decision  to  drop  a  design.  The  costs 
associated  with  the  reform  were  far  greater  than  the  fees 
paid  to  design  teams.  In  fact,  those  fees  made  up  less  than 
one-third  of  the  schools'  implementation  costs.  Significant 
costs  were  also  borne  by  teachers  in  terms  of  their  time 
and  effort  in  implementing  these  reforms. 

Overview  of  Student  Outcome  Findings 

Our  analysis  of  performance  trends  across  the  set  of 
schools  three  years  into  scale-up  focused  on  whether  NAS 
schools  made  any  gains  in  test  scores  relative  to  their 


respective  jurisdictions.  While  these  school-level  measures 
allowed  us  to  compare  performance  in  NAS  schools  with 
that  of  the  district  as  a  whole,  they  are  subject  to  signifi¬ 
cant  limitations.  For  example,  the  aggregated  measures 
may  fail  to  capture  changes  in  the  tails  of  a  distribution  or 
may  miss  some  significant  achievement  effects  that  would 
be  captured  if  longitudinal  student-level  data  were  avail¬ 
able  and  comparable  across  jurisdictions.  In  terms  of  stu¬ 
dent  outcomes,  we  found  the  following: 

•  Among  the  four  jurisdictions  with  ten  or  more  of  its 
schools  implementing  NAS,  Memphis  and  Kentucky 
appeared  to  be  the  most  successful  in  terms  of 
improvement  in  mathematics  test  scores,  whereas 
Cincinnati  and  Washington  schools  did  better  in  read¬ 
ing  scores. 

•  In  total,  of  the  163  schools  for  which  we  had  data  allow¬ 
ing  comparisons  in  performance  relative  to  the  district 
or  state,  81  schools  (50  percent)  made  gains  in  mathe¬ 
matics  relative  to  the  district  and  76  schools  (47  percent) 
made  gains  in  reading  relative  to  the  district. 

Because  of  the  wide  variation  in  implementation  that 
occurred  and  the  variety  of  environments  that  exist  within 
schools  and  among  jurisdictions,  it  may  have  been  too 
early  to  expect  robust  performance  results  across  the  NAS 
sites.  Better  and  longer-term  performance  data  are  needed 
in  order  to  make  conclusive  judgments  about  the  various 
designs  and  their  effects  on  school  performance. 

However,  our  implementation  analysis  showed  that 
the  overall  level  of  implementation  increased  modestly 
over  time,  and  there  was  continuing  within-school  varia¬ 
tion  in  implementation.  If  the  NAS  approach  to  school 
improvement  is  correct,  limited  implementation  will  lead 
to  weak  effects  on  student  performance.  Our  findings  sug¬ 
gest  that  we  cannot  expect  to  have  stronger  performance 
results  unless  implementation  significantly  increases. 

The  detailed  classroom  study  of  the  San  Antonio  juris¬ 
diction  allowed  us  to  examine  whether  variation  in 
instructional  conditions  was  related  to  student  achieve¬ 
ment,  when  controlling  for  other  student,  teacher,  class¬ 
room,  and  school  characteristics.  Our  findings  were  simi¬ 
lar  to  those  of  other  analyses  of  schoolwide  reforms: 

•  Strong  leadership  by  a  principal  as  reported  by  teachers 
had  significant  positive  effects  on  students'  state  test 
scores  in  reading  and  mathematics. 

•  Instructional  conditions  promoted  by  reforms  such  as 
NAS — including  teacher-reported  collaboration,  qual¬ 
ity  of  professional  development,  and  reform-like 
instructional  practices — were  not  related  to  student 
achievement,  net  of  other  student  and  classroom  con¬ 
ditions. 
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•  In  general,  early  implementation  of  NAS  designs  in  a 
high-poverty  district  that  exists  within  a  high-stakes 
accountability  system  did  not  result  in  significant  effects 
on  student  achievement. 

CONCLUSIONS 

In  the  end,  the  set  of  RAND  analyses  illuminated  three 
important  areas.  First,  the  analyses  provided  evidence  that 
an  external  agent  such  as  NAS  could  create  and  promote 
design  teams.  Second,  the  analyses  indicated  that  some  of 
the  theory  of  action  behind  the  NAS  efforts  was  under¬ 
developed,  and  they  pointed  to  important  conditions  for 
implementation  and  improved  student  outcomes.  Third, 
the  analyses  offered  important  lessons  on  how  to  carry  out 
future  efforts  at  reform. 

The  Contribution  of  New  American  Schools  to 
Educational  Reform 

NAS  did  accomplish  several  of  the  goals  it  had  set  for 
itself  and  in  the  process  made  a  number  of  important  con¬ 
tributions  to  educational  reform  that  should  be  kept  in 
mind.  These  include: 

•  NAS  funding  and  leadership  led  to  the  deliberate  devel¬ 
opment  of  several  functioning  whole-school  design 
teams. 

•  NAS  showed  that  external  change  agents  that  depended 
initially  on  external  funding  could  be  moved  toward 
self-sufficiency  over  time. 

•  NAS  explicitly  sought  scale-up  of  the  reform  initiative. 

•  NAS  actions  as  a  change  agent  have  significantly  influ¬ 
enced  policy  in  its  areas  of  interest. 

•  NAS  explicitly  made  analysis  of  design  implementation 
and  good  consumer  education  a  part  of  its  efforts. 

NAS’s  Theory  of  Action 

The  RAND  findings  provide  mixed  evidence  to  sup¬ 
port  NAS's  theories  of  positively  changing  schools  with 
whole-school  designs. 

•  The  initial  h\/pothesis  —  that  a  school  could  improve  its  perfor¬ 
mance  by  adopting  a  ivhole-school  design— zvas  largely 
unsupported.  Our  general  findings  showed  difficulties 

in  design  implementation  and  a  lack  of  significant 
improvement  in  school  performance  in  a  large  percent¬ 
age  of  the  schools  in  our  samples, 

•  The  hypothesis  that  the  designs  alone  ivere  not  helpful  to 
schools  and  that  schools  needed  assistance  in  implementation 
zvas  proven  correct.  Teachers  and  school  administrators 
clearly  reported  higher  levels  of  implementation  associ¬ 
ated  with  strong  assistance  from  design  teams.  Just  as 
important,  and  consistent  with  the  implementation  liter¬ 
ature,  conditions  within  the  schools  and  districts  and  the 


manner  of  selection  also  proved  to  be  important  to 
implementation  and  outcomes. 

•  The  scale-up  hypothesis  — that  a  district  that  converted  30 
percent  of  its  schools  using  zohole-school  approaches  zvould 
become  high  performing  and  not  revert  to  unproductive  prac¬ 
tices— zvas  disproved.  Districts  such  as  Memphis  quickly 
reverted  to  their  former  status  after  changes  in  their 
administrations. 

•  The  scale-up  hypothesis— that  a  district  needs  to  provide  a 
supportive  environment —zvas  dramatically  proven.  Barriers 
to  implementation  reported  by  school  staff  focused  on 
unsupportive  district  practices.  They  also  pointed  to  the 
challenges  inherent  in  simultaneously  adopting  multiple 
reforms,  high-stakes  testing  regimes,  and  new  school 
designs. 

•  In  general  loe  conclude  that  NAS's  initial  theory  of  action 
zvas  largely  underdeveloped  and  nonspecific  in  terms  of  ensur¬ 
ing  a  successful  scale-up  experience.  The  causal  chain  of 
events  leading  to  a  strong  implementation  and  positive 
outcomes  has  proven  to  be  far  more  complex  than  what 
was  originally  envisioned  by  NAS  and  one  that  remains 
largely  outside  of  NAS's  control  and  influence.  This 
finding  is  in  keeping  with  the  literature  on  implementa¬ 
tion  indicating  the  complexity  of  the  change  process. 

Implications  for  Future  Efforts 

Based  on  our  experience  with  NAS,  we  offer  the  fol¬ 
lowing  observations  for  any  future  attempts  at  scaling  up 

school  reforms. 

•  Externally  developed  education  reform  interventions  cannot 
"break  the  mold"  and  still  be  marketable  and  be  able  to  impile- 
rnented  in  current  district  and  school  contexts.  Schools  are 
not,  by  and  large,  fertile  ground  for  "break  the  mold" 
ideas,  often  because  of  a  lack  of  capacity  or  because  of 
local,  state,  or  district  regulations. 

•  External  interventions  need  to  address  systemic  issues  that 
can  hinder  implementation,  such  as  lack  of  teacher  capa¬ 
city  to  undertake  implementation  of  the  designs,  lack 
of  leadership  from  the  school  principal,  or  lack  of  a 
coherent  district  infrastructure. 

•  A  rush  to  scale-up  zvhcn  interventions  are  not  cofupdetely 
developied  zveakens  the  results.  This  problem  is  likely  to 
persist  partly  because  developers  are  under  financial 
pressure  to  scale  up  their  interventions  before  they  are 
thoroughly  evaluated  and  partly  because  districts  and 
schools  are  under  extreme  political  pressure  to  adopt 
external  solutions,  proven  or  not,  as  a  means  of  address¬ 
ing  the  lackluster  performance  of  their  students. 

•  Key  compionents  of  successful  implementation  are  consistent, 
clear,  andffvquent  communication  and  assistance  betzveen 
design  developyers  and  the  schools,  particularly  the  schools' 
teachers. 
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•  Monitoring  site  progress,  self-evaluation,  and  assessment  of 
outcomes  are  necessary  if  external  developers  are  to  be  success¬ 
ful  and  improve  their  offerings  over  time. 

•  The  typical  outcome  measures  used  in  public  accountability 
systems  provide  very  limited  indicators  of  student  and  school 
performance.  The  use  of  state-  and  district-mandated 
assessments  is  likely  to  present  a  constant  hindrance  to 
understanding  the  effect  of  innovative  approaches 
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